Systems and Methods of Building and Using an Image Catalog

ABSTRACT

A method manages an image catalog at a server system. The system receives reduced-resolution versions of one or more images stored in an image database of an external service distinct from the system. For each received reduced-resolution version, the system creates an index entry in the image catalog. The system receives a query from a user and matches the query to an index entry in the catalog, which corresponds to an image stored as a full-resolution version in the image database. The system requests authorization from the owner of the image. When authorization is received, the system retrieves the full-resolution version from the image database, and temporarily stores the full-resolution version in temporary storage. The system then transmits the full-resolution version of the image to the user and releases the full-resolution version of the image from the temporary storage in response to the transmitting the full-resolution version.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/941,502, entitled “Systems and Methods of Building and Using an ImageCatalog,” filed Nov. 13, 2015, which claims priority to U.S. ProvisionalPatent Application No. 62/080,198, “Systems and Methods of Building andUsing an Image Catalog,” filed Nov. 14, 2014, each of which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The disclosed implementations relate generally to digital imageapplications, and more particularly, to a digital image system forcataloging images from multiple sources and using the catalog fordiscovering and viewing images.

BACKGROUND

Image collections are growing in size and are often in multiplelocations. Image repositories may exist on local storage for mobile anddesktop devices, dedicated network-attached storage (NAS), or on remotecloud services. Often images are duplicated and stored in multiplerepositories because of the requirements of legacy tools that each wantcopies of your data in order to do their work. Social services, such asFacebook, Google+, and Twitter, are some of the largest cloud imagerepositories. It is becoming increasingly difficult to know the locationand history of a given image. Without adequate location and historyinformation of images, it is also difficult to conduct searches andprovide an intuitive visualization of search results. Conventionalmethods attempt to move images into a central location and performsearches against the pooled images. These conventional methods areinefficient and create security risks. For example, moving image datacreates risk in transit and exposes additional copies of the images tomisappropriation.

SUMMARY

Disclosed implementations address the above deficiencies and otherproblems associated with managing images. The present disclosure isdirected towards a system that provides visual insight, discovery, andnavigation into collections of millions of images stored in multiplelocations without needing to change a user's workflow. Search across theuser's entire portfolio using keywords and semantic informationautomatically extracted from the user's photos. In some implementations,the “asset-light” model does not move the user's image data (the actual“pixels”) from the cloud or local storage. The system thus allowsexploring the user's images to quickly finding exactly what the userneeds wherever the full-resolution images live. The system furthervisualizes the user's entire portfolio by graphing information about allof the user's images. In addition, the system uses statistical data toprovide business intelligence to bring value to the user's entireportfolio of images.

Some implementations provide simple and intuitive interfaces. Byleveraging leading-edge image processing and vision tools, the systemlets users perform simple searches and generate gorgeous, interactivegraphical representations of the users' data. Some implementations applythe latest Computer Vision algorithms to images in order to extractmetadata from the images. Such Computer Vision algorithms may include: aDeep Convolutional Neural Network to extract keywords; Optical CharacterRecognition to extract text, jersey numbers, signs, logos, and othercharacter-based information; Facial Recognition to match faces to names;Color Analysis and structural information (SIFT) from the images toidentify cropped and modified (Template Matching) images to trackduplicates and variations of images after processing. The systemextracts the existing metadata for each image, including its origin,dates, and statistical information. The system then compresses themetadata along with a deep image analysis and stores this data in acompact database designed to enable fast searches through truly enormousimage collections consisting of millions of individual images.

Some implementations include an image catalog that is orders ofmagnitude smaller than the actual images. By decoupling the search datafrom the geometrically expanding storage requirements for actual imagesand video data, the lightweight structure provides a computational layerthat represents all user images. Unlike other services, which forceusers to relocate their images, a system according to implementations ofthe present disclosure lets the images stay where they are now, avoidingchanges to the existing workflows and providing access to all the imagesin users' entire organization, rather than just the ones relocated to aparticular cloud service. Since the actual image data is not required tomove from its original location, the present disclosure provides a moresecure environment, especially because everything is encrypted andsecured.

As noted above, image collections are growing in size and are often inmultiple locations. It is becoming increasingly difficult to know all ofthe locations and history of a given image. The present disclosureaddresses this problem and allows users to manage the growing imagecollection complexity by indexing all of the users' image portfolios,finding duplicates and treating them as different versions of the sameoriginal image. Image analysis can even identify many imagemodifications, including color adjustments and cropping, provide animage history that shows all the variations and locations of a givenimage across all of the users' image collections and social services.

Systems, methods, devices, and non-transitory computer readable storagemedia for building and using an image catalog are disclosed. In someimplementations, a method of managing an image catalog is performed byone or more servers, each having one or more processors and memory. Themethod includes receiving from a first user identification of one ormore images in a first image database. The first image database isdistinct from the one or more servers. For each image of the one or moreimages, the method analyzes the respective image to extract respectivekeywords that describe the respective image and creates a respectiveindex entry in the image catalog. The respective index entry includesthe respective keywords. The method subsequently receives a query from asecond user and matches the query to one or more first index entries inthe image catalog. The first index entry corresponds to a respectivefirst image in the first image database. The method determines whetherthe second user is authorized to view the first images. When the seconduser is authorized to view one or more of the first images, the methodretrieves the corresponding first images from the first image databaseand transmits the retrieved first images to the second user.

In some implementations, analyzing the respective image to extractrespective keywords includes performing deep convolutional neuralnetwork semantic analysis on the respective image. In someimplementations, the first user specifies a subject matter for the oneor more identified images, and the deep convolutional neural networksemantic analysis uses a neural network trained on images correspondingto the specified subject matter.

In some implementations, analyzing the respective image to extractrespective keywords includes using optical character recognition (OCR),extracting a color palette for the respective image, identifying one ormore known faces in the respective image, and/or identifying one or moreknown human bodies based on body features, among others.

In some implementations, for each image, the method further includesidentifying metadata associated with the respective image. Therespective metadata includes one or more of: date/time the respectiveimage was created, location where the image was created, identificationof a camera that took the respective image, and identification of cameraattributes that took the respective image. The method stores therespective metadata as part of the respective index entry for therespective image.

In some implementations, a computer system for managing an image catalogincludes one or more processors, memory, and one or more programs storedin the memory. The programs are configured for execution by the one ormore processors. The programs include instructions for receiving from afirst user identification of one or more images in a first imagedatabase. The first image database is distinct from the one or moreservers. For each image of the one or more images, the programs analyzethe respective image to extract respective keywords that describe therespective image and create a respective index entry in the imagecatalog. The respective index entry includes the respective keywords.The programs receive a query from a second user and match the query to afirst index entry in the image catalog. The first index entrycorresponds to a first image in the first image database. The programsdetermine whether the second user is authorized to view the first image.When the second user is authorized to view the first image the programsretrieve the first image from the first image database and transmit thefirst image to the second user.

In some implementations, a non-transitory computer readable storagemedium stores one or more programs configured for execution by one ormore processors of a computer system for managing an image catalog. Theprograms include instructions for receiving from a first useridentification of one or more images in a first image database. Thefirst image database is distinct from the one or more servers. For eachimage of the one or more images the programs analyze the respectiveimage to extract respective keywords that describe the respective imageand create a respective index entry in the image catalog. The respectiveindex entry includes the respective keywords. The programs receive aquery from a second user and match the query to a first index entry inthe image catalog. The first index entry corresponds to a first image inthe first image database. The programs determine whether the second useris authorized to view the first image. When the second user isauthorized to view the first image the programs retrieve the first imagefrom the first image database and transmit the first image to the seconduser.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computing system for managing an imagecatalog in accordance with some implementations.

FIG. 2 is a block diagram of a server system in accordance with someimplementations.

FIG. 3 is a block diagram of a curator device in accordance with someimplementations.

FIG. 4 is a block diagram of a viewer device in accordance with someimplementations.

FIG. 5A is a process flow diagram illustrating establishing an imagecatalog with the assistance of a curator device in accordance with someimplementations.

FIG. 5B is a graphical user interface of a system tray with a curatorapplication icon displayed in accordance with some implementations.

FIG. 5C is a graphical user interface of a curator application inaccordance with some implementations.

FIG. 5D illustrates an example color spatial converted from an image inaccordance with some implementations.

FIG. 5E illustrates an example thumbnail color palette converted from animage in accordance with some implementations.

FIG. 5F is a process flow diagram illustrating a template matchingprocess in accordance with some implementations.

FIG. 6A is a graphical user interface of a navigator dashboard inaccordance with some implementations.

FIG. 6B is a graphical user interface of graph bar categories used inkeyword searches in accordance with some implementations.

FIG. 6C is an example map view in accordance with some implementations.

FIG. 6D is an example heatmap on a map view in accordance with someimplementations.

FIG. 7 is a process flow diagram illustrating a catalog transfer requestin accordance with some implementations.

FIGS. 8A and 8B provide a flowchart of a process for managing an imagecatalog in accordance with some implementations.

Like reference numerals refer to corresponding parts throughout thedrawings.

DESCRIPTION OF IMPLEMENTATIONS

Reference will now be made to various implementations, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the invention and the describedimplementations. However, the invention may be practiced without thesespecific details. In other instances, well-known methods, procedures,components, and circuits have not been described in detail so as not tounnecessarily obscure aspects of the implementations.

Disclosed implementations enable users to build a catalog and use thecatalog to provide visual insight, discovery and navigation intocollections of millions of images stored in multiple locations withoutneeding to change a user's workflow. In some implementations, the systemincludes at least two tools. The first one runs on the desktop and asecond one runs on the desktop, mobile, and/or in browsers. The firstcurator tool is responsible for seeking out images from local and cloudstorage, performing image analysis and uploading the results to an imagecatalog search database. The second viewer tool provides search,visualization, tagging, and delivery using the data stored in the imagesearch database. Image requests for locally stored images are fulfilledasynchronously by the curator tool. Both the curator and viewer toolsdisplay notifications of changes or global system messages. Systems anddevices implementing the curator and viewer tools in accordance withsome implementations are illustrated in FIGS. 1-4.

FIG. 1 is a block diagram of a computer system 100 that enables users tobuild and use an image catalog in accordance with some implementations.In some implementations, the computer system 100 includes viewer deviceapplication (or “module”) 102-1, 102-2 . . . executed on viewer devices104-1, 104-2 . . . , at least one curator user device 130, andserver-side software module 106 executed on a server system 108. Aviewer module 102 communicates with a server-side module 106 through oneor more networks 110. The viewer module 102 provides viewerfunctionality (e.g., search and view images) and communications withserver-side module 106. The server-side module 106 provides server-sidefunctionality (e.g., managing the image catalog 114 and handlingrequests to transfer images) for any number of viewer modules 102 eachresiding on a respective viewer device 104.

In some implementations, the viewer devices 104 are computing devicessuch as desktops, laptops, mobile devices, from which users 124 canbrowse the image catalog, discover images, and view images. The server108 connects to the external services 122 to obtain the images inresponse to an image uploading request initiated from the curator device130. The image uploading request may direct the curator device 130 toseek out images from local and cloud storage 123-1 . . . 123-N,performing image analysis and uploading the results to search database(e.g. server side image catalogs 114). The viewer modules 102 on theclient device 130 can then provide search, visualization, tagging, anddelivery using the data stored in the image catalog database 114. Imagerequests for locally stored images can be fulfilled asynchronously bythe curator device 130. Both the curator device 130 and the viewerdevice 104 are capable of displaying notifications of changes or globalsystem messages.

The computer system 100 shown in FIG. 1 includes both a client-sideportion (e.g., the viewer module 102 and modules on the curator device130) and a server-side portion (e.g., the server-side module 106). Insome implementations, data processing is implemented as a standaloneapplication installed on viewer device 104. In addition, the division offunctionality between the client and server portions of clientenvironment data processing can vary in different implementations. Forexample, in some implementations, the viewer module 102 is a thin-clientthat provides only image search requests and output processingfunctions, and delegates all other data processing functionality to abackend server (e.g., the server system 108). In some implementations,the curator device 130 delegates the image analysis function to abackend server (e.g., the server system 108).

The communication network(s) 110 can be any wired or wireless local areanetwork (LAN) and/or wide area network (WAN), such as an intranet, anextranet, or the Internet. It is sufficient that the communicationnetwork 110 provides communication capability between the server system108 and the clients 104, and the curator device 130.

In some implementations, the server-side module 106 includes one or moreprocessors 112, one or more databases 114, an I/O interface to one ormore clients 118, and an I/O interface to one or more external services120. The I/O interface to one or more clients 118 facilitates theprocessing of input and output associated with the client devices anddevices for server-side module 106. One or more processors 112 obtainimages and information related to images from external services 122 inresponse to an upload request initiated by the device 130, process theimages and the information, and store the image references along withthe information in the image catalog 114. The image catalog database 114stores various information, including but not limited to catalogs,images, image metadata, image information, geographic information, mapinformation, among others. The image catalog 114 may also store aplurality of record entries relevant to the users associated withimages. I/O interface to one or more external services 120 facilitatescommunications with one or more external services 122 (e.g., imagerepositories, social services, and/or other cloud image repositories).

In some implementations, the server-side module 106 connects to theexternal services 120 through the I/O interfaces 120 and obtaininformation such as images stored on the external services 120. Afterobtaining the images along with the information associated with theimages, the server 108 processes the data retrieved from the externalservices 120 to extract information and catalog the images. Theprocessed and/or the unprocessed information are stored in the imagecatalog 114, including but not limited to catalogs, images, imagemetadata, image information, geographic information, map information,among others. The database 114 may also store a plurality of recordentries relevant to the users associated with location sharing, andshort electronic messages exchanged among the users.

Examples of the viewer device 104 include, but are not limited to, ahandheld computer, a wearable computing device, a personal digitalassistant (PDA), a tablet computer, a laptop computer, a cellulartelephone, a smart phone, an enhanced general packet radio service(EGPRS) mobile phone, a media player, a navigation device, a portablegaming device console, a tablet computer, a laptop computer, a desktopcomputer, or a combination of any two or more of these data processingdevices or other data processing devices.

The viewer device 104 includes (e.g., is coupled to) a display and oneor more input devices. The viewer device 104 receives inputs (e.g.,messages, images) from the one or more input devices and outputs datacorresponding to the inputs to the display for display to the user 124.The user 124 uses the viewer device 104 to transmit information (e.g.,messages, images, and geographic location of the viewer device 104) tothe server 108. The server 108 receives the information, processes theinformation, and sends processed information to the display of theviewer device 104 for display to the user 124.

Examples of the curator device 130 include, but are not limited to, ahandheld computer, a tablet computer, a laptop computer, a desktopcomputer, a server computer, or other computing device with sufficientprocessing power to operate as a server, or a combination of any two ormore of these data processing devices or other data processing devices.The curator device 130 includes (e.g., is coupled to) a display and oneor more input devices in some implementations. The curator device 130receives inputs (e.g., requests to upload or retrieve images) from theone or more input devices and outputs data corresponding to the inputsto the display for display to the user 132. The user 132 uses thecurator device 130 to transmit information (e.g., requests to upload,search, and retrieve images) to the server 108. The server 108 receivesthe information, processes the information, and sends processedinformation (e.g., uploading status, search results) to the display ofthe device 130 for display to the user 132.

Examples of one or more networks 110 include local area networks (LAN)and wide area networks (WAN) such as the Internet. One or more networks110 are, optionally, implemented using any known network protocol,including various wired or wireless protocols, such as Ethernet,Universal Serial Bus (USB), FIREWIRE, Global System for MobileCommunications (GSM), Enhanced Data GSM Environment (EDGE), codedivision multiple access (CDMA), time division multiple access (TDMA),Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or anyother suitable communication protocol.

The server system 108 is implemented on one or more standalone dataprocessing apparatuses or a distributed network of computers. In someimplementations, the server system 108 also employs various virtualdevices and/or services of third party service providers (e.g.,third-party cloud service providers) to provide the underlying computingresources and/or infrastructure resources of the server system 108.

The computer system 100 shown in FIG. 1 includes both a client-sideportion (e.g., the viewer module 102, modules on the device 130) and aserver-side portion (e.g., the server-side module 106). In someimplementations, a portion of the data processing is implemented as astandalone application installed on the viewer device 104 and/or thecurator device 130. In addition, the division of functionality betweenthe client and server portions of client environment data processing canvary in different implementations. For example, in some implementations,the viewer module 102 is a thin-client that provides user-facing inputand output processing functions, and delegates data processingfunctionality to a backend server (e.g., the server system 108). Inanother example, in some implementations, the curator device 130delegates data processing functionality (e.g., image analysis) to abackend server (e.g., the server system 108).

FIG. 2 is a block diagram illustrating the server system 108 inaccordance with some implementations. The server system 108 may includeone or more processing units (e.g., CPUs and/or GPUs) 112, one or morenetwork interfaces 204 (e.g., including an I/O interface to one or moreclients 118 and an I/O interface to one or more external services 120),one or more memory units 206, and one or more communication buses 208for interconnecting these components (e.g. a chipset).

The memory 206 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 206, optionally, includes one or morestorage devices remotely located from one or more processing units 112.The memory 206, or alternatively the non-volatile memory within thememory 206, includes a non-transitory computer readable storage medium.In some implementations, the memory 206, or the non-transitory computerreadable storage medium of the memory 206, stores the followingprograms, modules, and data structures, or a subset or superset thereof:

-   -   operating system 210 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   network communication module 212 for connecting server system        108 to other computing devices (e.g., viewer devices 104 and        external service(s) 122) connected to one or more networks 110        via one or more network interfaces 204 (wired or wireless);    -   server-side module 106, which provides server-side data        processing (e.g., user account verification, instant messaging,        and social networking services), includes, but is not limited        to:        -   an image source module 222 for connecting to image source            repositories (i.e. image database 123 on external service            122);        -   an image analysis module 224 that processes images and            stores the processed images along with image metadata to the            image catalog 114;        -   a transfer module 226 that communicates with the curator            device 130 and the viewer device 104 to handle transfer            requests;        -   a notification module 228 to send and receive notifications            to and from the viewer device 104 and the curator device            130;        -   a status module 230 to provide status of the images during            transfer; and        -   a catalog module 232 to manage the image catalog 114.    -   one or more server database of image catalogs 114 storing data        for image catalogs, including but not limited to:        -   an image identifier 242 to uniquely identifies each image            stored in the image catalog 114; in some implementations,            each image has a globally unique identifier (GUID); the            image catalog 114 can be queried to return a list of GUIDs            using standard query language, and group requests are            structured using either query language or an array of GUIDs;        -   an image metadata 244 storing metadata extracted from images            processed by the image analysis module 228;        -   a thumbnail 246 storing thumbnail of images; the image            catalog 114 caches results locally on demand.        -   a source repository reference 248 storing a reference to the            location of the full size images stored in the source            repositories (e.g., the image databases 123 in the external            services 122); and        -   though not shown in FIG. 2, a temporary container to            temporarily store full size images during image transfer.

In some implementations, the image catalog module 232 manages multipleimage repositories, providing methods to access and modify data that canbe stored in local folders, NAS or cloud-based storage systems. Theimage catalog 114 can even search offline repositories. Offline requestsare handled asynchronously, with large delays or hours or even days ifthe remote machine is not enabled. The image catalog module 232 managespermissions and secure access for a wide range of databases. Queries tothe image catalog 114 work on groups of images to reduce bandwidth andimprove performance in accordance with some implementations. Each of thedata types including but not limited to the image identifier 242, theimage metadata 244, the thumbnail 246, and/or the source repositoryreference 248 can be requested for an entire group of images in a singlecall.

Though not shown in FIG. 2, one of more caches are used to improveperformance. In some implementations, a local NoSQL database is used tocache results in a format suitable for fast in-memory access. Inparticular, thumbnails 246 are cached separately from metadata 244 forimproved UI performance. In some implementations, thumbnails 246 arestored in standard 4-bit PNG files using their QUID as a filename,making debugging easier.

In some implementations, the data associated with each image can bebroken into three groups, the original metadata as part of the imagemetadata 244, a computed thumbnail 246, and the metadata generatedthrough image analysis to aid search, also stored as part of the imagemetadata 244. In some implementations, the thumbnail 246 typically uses2-3 KB, the original metadata ranges from 0-1 KB, and the analyzedmetadata is only a few hundred bytes. The analyzed metadata is the mostfrequently accessed data, and it is often cached locally in accordancewith some implementations. Thumbnails 246 are also cached to disklocally in accordance with some implementations, so that the in-memoryfootprint of the application is minimized. The original metadata iscompressed using context-specific knowledge in accordance with someimplementations. For example, commonly used parameters like f-stop,manufacturer's name, or keywords can be stored using single-byte valueswhich are automatically expanded as needed. This reduces the metadatasize to a couple hundred bytes. In another example, storing compressedkeywords and commonly searched values as single-integer values alsoimproves search performance significantly by replacing stringcomparisons with integer comparisons.

In addition to storing metadata for images, in accordance with someimplementations, the present invention can be extended to handle videos.For example, the image analysis process described in further detailsbelow can be performed on each frame of a video to compute keywords froma Deep Convolutional Neural Network, a color palette, histogram, OCR,and/or facial recognition, among others. After the image analysis, eachmetadata field stored in the image metadata 244 can include a framerange. During video analysis, efficient video processing can includeskipping frames, reducing resolution, and/or using video compressionhints, among others. For example, metadata associated with every N(N>=1) frames are analyzed and compared against previously computedvalues. If a value is added, the first frame of its range is marked andstored in the image metadata 244, and if a value is deleted the end ofthe frame range is marked and stored in the image metadata 244. Thisenables search within video sequences using a compact metadatarepresentation.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, memory206, optionally, stores a subset of the modules and data structuresidentified above. Furthermore, memory 206, optionally, stores additionalmodules and data structures not described above.

FIG. 3 is a block diagram illustrating a representative curator device130 in accordance with some implementations. A curator device 130,typically, includes one or more processing units (e.g., CPUs and/orGPUs) 302, one or more network interfaces 304, memory 306, optionallyone or more sensors or image caption devices, and one or morecommunication buses 308 for interconnecting these components (sometimescalled a chipset). Curator device 130 also includes a user interface310. The user interface 310 includes one or more output devices 312 thatenable presentation of media content, including one or more speakersand/or one or more visual displays. The user interface 310 also includesone or more input devices 314, including user interface components thatfacilitate user input such as a keyboard, a mouse, a voice-command inputunit or microphone, a touch screen display, a touch-sensitive input pad,a camera, a gesture capturing camera, or other input buttons orcontrols. Furthermore, some curator devices 104 use a microphone andvoice recognition or a camera and gesture recognition to supplement orreplace the keyboard.

Memory 306 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 306, optionally, includes one or more storagedevices remotely located from one or more processing units 302. Memory306, or alternatively the non-volatile memory within memory 306,includes a non-transitory computer readable storage medium. In someimplementations, memory 306, or the non-transitory computer readablestorage medium of memory 306, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   operating system 316 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   network communication module 318 for connecting the curator        device 130 to other computing devices (e.g., the server system        108 and external service(s) 122) connected to one or more        networks 110 via one or more network interfaces 304 (wired or        wireless);    -   presentation module 320 for enabling presentation of information        (e.g., a user interface for a social networking platform,        widget, webpage, game, and/or application, audio and/or video        content, text, and/or displaying an encoded image for scanning)        at the curator device 130 via one or more output devices 312        (e.g., displays, speakers, etc.) associated with user interface        310;    -   input processing module 322 for detecting one or more user        inputs or interactions from one of the one or more input devices        314 and interpreting the detected input or interaction (e.g.,        processing the encoded image scanned by the camera of the client        device);    -   one or more applications 326-1-326-N for execution by the        curator device 130 (e.g., camera module, sensor module, games,        application marketplaces, payment platforms, social network        platforms, image source module, image analysis module, transfer        module, notification module, status module, catalog module,        and/or other applications involving various user operations);    -   curator module, which provides curator data processing and        functionality, including but not limited to:        -   image source module 332 for connecting to image source            repositories (i.e. image database 123 on external service            122);        -   image analysis module 334 that processes images and stores            the processed images along with image metadata to the image            catalog 114;        -   transfer module 336 that communicates with the curator            device 130 and the viewer device 104 to handle transfer            requests;        -   notification module 338 to send and receive notifications to            and from the viewer device 104 and the server 108;        -   status module 340 to provide status of the images during            transfer; and        -   catalog module 342 to remotely manage the image catalog 114.

In some implementations, the image analysis module 334 uses a rule-basedlanguage to allow custom extensions of the analysis that incorporatescustom decision rules and external data sources, such as spreadsheets ordatabases. For example, the data manager 132 can write a rule that looksup baseball statistics from a company database based on the dateassociated with the image. These custom rules can be run once by theimage analysis module 334 and the results can be cached as keywordsstored in the image catalog 114 as the image metadata 244.

In some implementations, the curator applications 326 runs as an OSXtoolbar service app, or as a Windows system tray application thatmonitors a set of local or remote image repositories (e.g., the imagedatabases 123), analyzes the images in the set of local or remote imagerepositories, and after the analysis, uploads image metadata for anyimages added (or removed) to the source image repositories (e.g., theimage databases 123). Referring back to FIG. 1, though FIG. 1illustrates one curator device 130, multiple curator devices 130 mayexist on separate devices working with a shared image catalog 114.

The image analysis performed by the image analysis module 334 includesextracting semantic information from the image using computer vision,OCR, facial recognition and color palette algorithms that can becompute-intensive. In some implementations, the image analysis isperformed when the system is idle so that normal desktop work is notdisrupted.

FIG. 4 is a block diagram illustrating a representative viewer device104 in accordance with some implementations. A viewer device 104,typically, includes one or more processing units (CPUs) 402, one or morenetwork interfaces 404, memory 406, optionally one or more sensorsand/or image capture devices, and one or more communication buses 408for interconnecting these components (sometimes called a chipset).Viewer device 104 also includes a user interface 410. The user interface410 includes one or more output devices 412 that enable presentation ofmedia content, including one or more speakers and/or one or more visualdisplays. The user interface 410 also includes one or more input devices414, including user interface components that facilitate user input suchas a keyboard, a mouse, a voice-command input unit or microphone, atouch screen display, a touch-sensitive input pad, a camera, a gesturecapturing camera, or other input buttons or controls. Furthermore, someviewer devices 104 use a microphone and voice recognition or a cameraand gesture recognition to supplement or replace the keyboard.

Memory 406 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 406, optionally, includes one or more storagedevices remotely located from one or more processing units 402. Memory406, or alternatively the non-volatile memory within memory 406,includes a non-transitory computer readable storage medium. In someimplementations, memory 406, or the non-transitory computer readablestorage medium of memory 406, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   operating system 416 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   network communication module 418 for connecting viewer device        104 to other computing devices (e.g., server system 108 and the        curator device 130) connected to one or more networks 110 via        one or more network interfaces 404 (wired or wireless);    -   presentation module 420 for enabling presentation of information        (e.g., a user interface for a social networking platform,        widget, webpage, game, and/or application, audio and/or video        content, text, and/or displaying an encoded image for scanning)        at viewer device 104 via one or more output devices 412 (e.g.,        displays, speakers, etc.) associated with user interface 410;    -   input processing module 422 for detecting one or more user        inputs or interactions from one of the one or more input devices        414 and interpreting the detected input or interaction (e.g.,        processing the encoded image scanned by the camera of the client        device);    -   one or more applications 426-1-426-N for execution by viewer        device 104 (e.g., search module, transfer request module,        visualization module, categorization module, delivery module,        notification module, camera module, sensor module, games,        application marketplaces, payment platforms, social network        platforms, and/or other applications involving various user        operations);    -   viewer module 102, which provides client-side data processing        and functionality, including but not limited to:        -   search module 432 to perform search of image(s) in the image            catalog 114;        -   transfer request module 434 to handle image transfer            requests initiated by the viewer 124;        -   visualization module 436 to present visualization of images            in a dashboard as shown in FIG. 6A to the viewer 124;        -   categorization module 438 to categorize images and manage            tags of images;        -   delivery module 440 to deliver images in various formats;            and        -   notification module 442 to send and receive notifications to            and from the server 108 and/or the curator device 130; and    -   client data 450 storing data of a user associated with the        viewer device 104, including, but is not limited to:        -   user profile data 452 storing one or more user accounts            associated with a user of viewer device 104, the user            account data including one or more user accounts, login            credentials for each user account, payment data (e.g.,            linked credit card information, app credit or gift card            balance, billing address, shipping address, etc.) associated            with each user account, custom parameters (e.g., age,            location, hobbies, etc.) for each user account, social            network contacts of each user account; and        -   user data 454 storing usage data of each user account on            viewer device 104.

FIG. 5 is a process flow diagram 500 illustrating image uploads by acurator device 130 in accordance with some implementations. In someimplementations, the curator device 130 is responsible for seeking outimages from local and cloud storage (e.g., the image databases 123),performing image analysis and uploading the results to the image catalogdatabase 114. In some implementations, the curator device 130 can managemultiple image repositories (e.g., image databases 123 in externalservices 122) by first connecting (502) to the image repositories. Insome implementations, a mechanism (e.g., a software development kit,SDK) is provided for the curator device 130 to add new databaseconnections to enable the addition of new and proprietary imagerepositories and to interact with proprietary systems. In someimplementations, the computer system 100 has a modular system for addingsupport for new database types. For example, a set of modules can beprovided to support a wide range of existing image databases 123,including Flickr, Google+, Dropbox, Facebook, Adobe Lightroom, and manyothers.

After a successful authentication to connect (502) to the imagerepositories (e.g., the image databases 123), the curator device 130seeks (504) images stored in the image repositories (e.g., the imagedatabase 123) and performs (506) image analysis. Many image fileformats, including JPG, TIFF, and PNG, can store textual informationalong side the image pixels. Methods in accordance with someimplementations access image metadata independently from the pixel data.Certain metadata fields can be accessed independently, without requiringthe transfer of the entire image metadata block. In someimplementations, metadata for groups of images can be accessed with asingle method call to minimize bandwidth. The returned data is encryptedand compressed during transfer to ensure security in accordance withsome implementations. In some implementations, the present disclosurecan be extended to applications used in the medical industry (e.g.,HIPPA, DIACOMP). The medical industry has a number of unique securityissues and specialized image file formats and metadata fields. Extendingthe core services in this present disclosure to handle these featuresfits well within the SDKs described in further details below.

The metadata 244 is generally extensible, allowing the storage ofarbitrary text that can include keywords, camera information, GPScoordinates, or program parameters. Due to the extensible nature of themetadata 244, the present disclosure does not rely on the existence ofany particular metadata field in all images when performing (506) imageanalysis. In some implementations, the curator device 130 extracts themetadata 244 from the source images stored in the image databases 123,and enhances it by running and performing (506) a series of imageanalysis operations to compute semantic and structural data about theimage before uploading (520) the results to the image catalog 114. Theuploaded image analysis result includes but not limited to metadata 244,thumbnail 246, and a reference to the source repository 248 (e.g., alink to the image location in the image database 123).

In some implementations, many of the image analysis (506) processes canoptionally use a graphic processing unit (GPU) to improve performance,if available. The image analysis performance (506) are expensive toexecute, often taking many seconds to run on each image. In someimplementations, the system performs this time-consuming work in thebackground on the desktop computer (e.g. the curator device 130 by theimage analysis module 334) or the server 108 running the image analysismodule (e.g. by the image analysis module 224). In some implementations,the server 108 receives from a first user identification (e.g., the datamanager 132) of one or more images in a first image database (e.g., theimage database 123), the first image database (residing on the externalservice 122) is distinct from the one or more servers, for each image ofthe one or more images, the server 108 analyzes the respective image toextract respective keywords that describe the respective image, andcreates a respective index entry in the image catalog 114, therespective index entry includes the respective keywords.

FIG. 5B is an example user interface illustrating a curator applicationthat runs in the background. The curator application shown as an icon510 monitors the curator device 130 and/or the server 108 usage andpauses processing so as not to block foreground applications running onthe curator device 130 and/or the server 108. In some implementations,the curator application runs as an OSX toolbar service app, or as aWindows system tray application that monitors a set of local or remoteimage repositories (e.g., the image databases 123), analyzes and uploadsimage metadata for any images added (or removed) to the repositories(e.g., the image databases 123). The image analysis performed (506) bythe curator device 130 as shown in FIG. 5A can be compute-intensive. Insome implementations, the image analysis is performed (506) when thecurator device 130 is idle, so that the normal curator device 130desktop work is not interrupted by the image analysis. In someimplementations, the curator device 130 delegates the image analysisfunction to a backend server (e.g., the server system 108).

FIG. 5C illustrates a portion of an example curator application userinterface. In some implementations, modules of the curator application(e.g., the status module 230 and/or the status module 340) calculatecurrent status 522 of the monitored repositories. And the interface tothe curator application displays the calculated current status 522 ofthe monitored repositories, showing the number of images uploaded, thenumber of pending images remaining to be analyzed along with atime-to-completion estimate. A data manager 132 can add or removerepositories by specifying local folders, or by providing authenticationlogin information for secured databases or cloud storage (e.g., theimage databases 123). A large range of remote services including bothconsumer services such as Google, Flickr and Facebook and enterpriseservices such as Box, Dropbox, and secured databases are supported. Aset of SDK enables the addition of repository types to enable addingproprietary image database formats.

Image analysis can be performed either on the full resolution imagedata, or on reduced resolution image thumbnails to improve performanceat the expense of some quality loss. The image analysis can load theappropriately-sized image from the source repository (e.g., the imagedatabases 123), perform a series of complex analysis that extractsemantic and structural information from the image. For example, duringthe vision categorization (508) process, a set of categories 522, orimage tags, are computed. In some implementations, the categories 522 orthe tags describe the contents of the image using Deep ConvolutionalNeural Network Computer Vision algorithms. Tags such as “sunset” or“building” can be computed by examining the image which are very usefulfor natural search by the viewer device 104.

As used herein, tags are strings that are stored in the image metadata244, such as in the IPTC:Keyword or EXIF UserComment fields. Tags can begenerated automatically by the curator application during the imageanalysis, or applied manually in the viewer tool or other image editingapplications. Tags are typically used to identify features or propertiesof an image. Tags are often used to group images into sets, such asMinter or #Landscape, and they can also be used to manage workflow, suchas #Draft or #Public.

The categories as tags, along with other information about the faces,text, colors and key visual structure of the image are then encodedalong with the native image metadata (EXIF, IPTC, dates, times, cameraand other information included in many image files) and uploaded assecurely encoded and compressed data to the image catalog 114. In someimplementations, the curator device 130 uploads about 2-5 KB of data perimage, whereas a full-resolution image file is often tens of megabytes,a 1000× compression ratio. The bulk of the uploaded data is a compressedimage thumbnail 246. The thumbnails 246 are used by the viewer module102 in lieu of loading the entire image from the image databases 123,which is not possible for some of the local image repositories.

In some implementations, compressed image data is less than 4K perimage. The compressed image data may include the metadata from theoriginal image, the additional metadata computed through image analysis.The compressed data may include a thumbnail image (e.g., used as aproxy). As a result, a user can store 1 MB image references for about 3GB of original image. This is often small enough so that the entireCatalog can be cached locally, even on mobile devices, to improveinteractivity and search performance dramatically.

Referring back to FIG. 5A, in some implementations, the image analysisperformance (506) provides a set of leading-edge image processingalgorithms that extract semantic and structural information from animage to improve search. These time-consuming and complex algorithms areperformed as part of the analysis by the curator device 130, requiringno user-intervention. The results are not directly apparent to the datamanager 132 and/or the user 124. Instead, the data manager 132 and/orthe user 124 is merely able to perform natural language and SQL-likesearch queries. Rather than manually tagging and organizing the images,image analysis is performed (506) to compute a cluster of high-qualitysemantic and structural information for each image completelyautomatically. Search algorithms then use this data to produce rankedresults providing smart, fully automatic image search.

The processes (508-518) are run on reduced-resolution versions of theoriginal image stored in the image databases 123 (unless the original issmall). In some implementations, the processes (508-518) include visioncategorization (508) (e.g. a Deep Convolutional Neural Network) toextract image categories (keywords), Optical Character Recognition (OCR)(510) to extract text, facial recognition (512) to find faces and matchthem up to known people, color analysis (514) and structural analysis(516) to identify similar images. The errors produced during theprocesses (508-510) can be reduced by training. The morecontext-specific training the system performs, the higher quality theresults.

When performing (506) image analysis, the latest computer visionalgorithms (vision categorization (508)) can extract meaningfulinformation from the images stored in the image databases 123. Computervision categorization (508) is also an auto-tagging process. In someimplementations, the categorization (508) uses a Deep ConvolutionalNeural Network to analyze each image to produce a set of keywordsdescribing the semantic content. For example, one can analyze vacationphotos and return keywords such as “sunset”, “building” or “trees” anduse these to improve the system's ability to search images with naturallanguage. The training can continue to improve the quality of the imageanalysis results. After ingesting, the system checks to see if any ofthe images being analyzed (506) contain manually added keywords, and thesystem uses those images to improve the training data set. That way theautomatic tagging process done using categorization can havecontext-specific improvements for each client. To further improve thequality of search results, in some implementations, the system storesthe keyword sets and associated error metrics returned by visioncategorization (508). The stored keyword sets and associated errormetrics can then be used to train and improve the quality of the imageanalysis (506) process.

OCR (510) can be used to extract text, jersey numbers, signs, and logos.The text extracted from each image is stored in a metadata field (e.g.,UserComment). During natural language searches, this metadata field canbe used as part of the search, because it often contains company names,street signs, jersey numbers, or other information (e.g., Starbucksstreet sign) that improves natural language search.

Facial recognition (512) can be used to match faces to names. The facialrecognition (512) is performed on all images and a training network canbe used to match facial geometry to known faces. The training set can beextended using manual facial identification from within the viewerdevice (104) tool. By identifying people in a few images, the facialrecognition (512) process can do a significantly better job of findingthose same people in other images. For example, after one identifies aCEO or key individuals in a handful of images, the system can use thatdata to find those same people in the other photos in the user'sportfolio.

Color analysis (514) and SIFT (516) can identify cropped and modifiedimages to track duplicates and variations of images after processing.The system extracts the metadata for each image including its origin,dates and statistical information and compress it along with a deepimage analysis and store this data in a custom database, the imagecatalog 114, that is designed to enable fast searches through trulyenormous image collections consisting of millions of individual images.

There are two forms of color analysis on images as shown in FIG. 5D andFIG. 5E. First, as shown in FIG. 5D, the image 530 is converted into a4-bit (16 color) thumbnail 532. And second, as shown in FIG. 5E, a 9color palette 542 is computed with spatial information. As used herein,color palette extraction determines the distribution of color in eachimage or distribution of color within designated sub-regions of eachimage. As shown in FIG. 5E, the 9-color spatial palette 542 is designedto be used as a thumbnail proxy, where each of the 9 colors isassociated with one of 9 regions in the original image 540, similar to a3×3 thumbnail. The color displayed for each of the regions represents anaverage of the colors from the corresponding region in the originalimage 540. Each region in the original image 540 is given a weight toindicate its relative size, so that one can vary the size of each regionto better match the original image while still using minimal storage.The 9-color palette 542 is embedded with the image metadata so that itcan be accessed quickly, without requiring download of the thumb image246.

As used herein, thumbnails 246 are small representations of images. Thesystem 100 assumes that each image has an associated thumbnail. In someimplementations, thumbnails 246 are stored using 4 bits per pixel, eachof which can take on one of 16 unique colors, with ˜10,000 pixels (e.g,100×100). A typical thumbnail takes 2-3 KB of data, compared to tens ofmegabytes for full resolution photos. In some implementations, the imageanalysis creates thumbnails for each new or modified image usinghigh-quality decimation filters and palette analysis to select the best16 colors. A 9-tile spatial palette 542, which selects nine colors fromthe image along with their spatial location and weights, is computed andstored as a special metadata field to use as a proxy when the full 4-bitthumbnail is not available, or when the onscreen thumbnail is only a fewpixels in size. This allows the 9-color palette 542 to be used as aquick proxy even when the image is displayed as only a few pixels onscreen. The 9-color spatial 542 and 16-color thumbnail palettes 532 areboth used to aid in search and ordering and graphing images based oncolor. Each color is given a priority weighting so that one can define auniquely ordered sort. This allows users to sort thumbnails by color, orplot images in a scatter diagram based on color usage.

The image analysis described above extracts semantic information fromthe image pixels. During vision categorization (508), algorithms such asa Deep Convolutional Neural Network, based on a proprietary trainingset, extract keywords describing the semantic objects in the image.Multiple independent neural networks with context dependent trainingsets provide expert analysis of specific types of images. Rather than asingle network trained for all images, the discrete networks eachoperate independently, providing a ranked set of keywords that identifygeneric objects, company specific identifiers (logos, colors, people),topic specific data (sports stadiums and locations, known paintings,medical imagery) and combine the results to provide a broad set ofcontext specific terms. The discrete networks accept high level dataincluding perceptual color spaces (e.g. HSV & LAB), subject basedsegmented color palettes, and shape analysis. During OCR (510), any textfrom the image including logos, street signs, jersey numbers, ordocument text, can be extracted. During facial recognition (512), basicfacial and body features are extracted and matched against a set ofknown geometries to generate names.

The users can further extend the semantic processing using a domainspecific language to define heuristic rules. Individual algorithms areenhanced in a second round of extensible rule based processing thatcombines the results of the individual steps with known global datasets.For example, if the execution of deep convolutional neural networkanalysis during the vision categorization (508) extracts the keyword“baseball”, the color analysis (514) can include a match against knownbaseball team colors to generate a team name. Users can implement theirown custom rules for their unique content and data. Context specificdata, such as company events, historical statistics (e.g. play-by-playdata from each game), facial geometries of key individuals, andcorporate hierarchies can be compared against the data generated by theinitial pass to generate a higher level set of keywords that improvesearch. For example, one can search for “triple plays in Dodger'sstadium” or “Stanley Cup Playoffs” neither of which is directlyaccessible via basic image analysis without context specific data.

Still referring to FIG. 5A, template matching (518) finds matchesbetween images using structural (geometric) analysis. For example, ifone crops an image, template matching (518) searches for a match betweenthe cropped region and all other images. Because template matchingrelies on the resolution independent features in the image, rather thanindividual pixel values, it can find matches between images that arescaled, color corrected, or even after moderate adjustments. Softermatches return results of images with similar layouts. When combinedwith categorization terms, the soft matches allow the users to dosearches for “similar” images.

The more exact matches allow users to find image duplicates, and totrack the history of an image through image adjustments (colorcorrection, editing, etc) and to find copies of the same image posted tovarious repositories and social services. One can combine these into asingle image “timeline” history, so that one can treat multiple copiesof the image as a single unit and know its entire version and postinghistory.

Template matching 518 detects when one image is extracted (cropped) fromanother image, or if two images share the same source. Thistime-consuming process can be used to identify images that were croppedfrom an original. Doing template matching on large image collections isprohibitively expensive when performed using brute-force matching (N₂).However, the system in accordance with some implementations candramatically reduce the number of comparisons using additional imagemetadata, dates, and locations etc.

FIG. 5F is a process flow illustrating the template matching (518)function in accordance with some implementations. During a curatordevice (130) and/or a server (108) first obtains (552) a set of firstimage metadata corresponding to a first image, followed by obtaining(554) a set of second image metadata corresponding to a second image forcomparison (556) of the set of first image metadata with the set ofsecond image metadata. In response to a substantial match of the set offirst image metadata and the set of second image metadata indicatingsimilar layouts between the first image and the second image, the systemlinks (558) the first image to the second image. For example, when thesystem detects a cropped, duplicated, or color corrected image based onthe metadata comparison, the system can link it as a variation theoriginal image so that the relationship can be displayed as part of ahistory view in the viewer application. To reduce the number ofcomparisons, in accordance with some implementations, the system can inresponse to detecting at least one item from the set of first imagemetadata and the set of second image metadata indicating the first imageand the second image do not match, skip (560) the rest of the comparisonof the set of first image metadata with the set of second imagemetadata. For example, using image creation dates in the metadata, thesystem can skip comparing an image created earlier than the test crop.In another example, the system can also do simple metadata matching tofind images with the same metadata, thus skipping commonly modifiedfields, like the creation date.

FIG. 6A is an example graphical user interface 600 illustrating anoutput on a viewer device 104. In some implementations, the viewermodule 102 provides a real-time dashboard for an entire image portfoliowith views for thumbnails 612, maps 614, and graphs 616 along with asearch bar 602 to allow a user to refine the set of images displayed ineach view. The viewer 124 can use the search bar 602 to quickly findimages that have been uploaded to the image catalog 114 from any of theimage repositories (e.g., the image databases 123). The search resultsdisplayed in the visualization dashboard by the visualization module 434can be dynamically updated. Most gallery applications provide a peepholeinto a huge collection, showing only a dozen or two image thumbnails atany moment. Using graphs 616 and maps 614, the viewer module 102provides an overview of large groups of images using interactivestatistical visualizations. The viewer 124 can view thousands ofthumbnails as tiny icons ordered into useful groups, perhaps months oryears, by color palettes or by categories.

When performing searches using the search bar 602, an image searchlibrary (e.g., the search module 432) uses the data computed by theimage analysis library (e.g., the image analysis module 224 and/or theimage analysis module 334) and stored in the image catalog 114 tocompute a ranked set of results for a given search string. The searchlibrary then breaks the search string entered in the search box 608 intoboolean commands and a number of specialized ranking metrics to sort theimages in the database, returning any that are considered high qualitymatches. The entirety of the each search is captured in a single string,displayed in the search box 608. In some implementations, search historyis a stack of search strings accessed via the back button 604 andforward button 606 adjacent to the search box 608, matching thewell-understood browser interface pattern. Searches can be triviallycopied and pasted, sent via email, or stored for later use or sharingwith colleagues in accordance with some implementations.

Like the image identifiers 242 stored in the image catalog 114, in someimplementations, images are identified using globally unique identifiers(GUIDs) and search results are arrays of GUIDs in some implementations.Each result includes a GUID and a rank to indicate the quality of thematch. The quality of all results is analyzed using clustering to selecta final search result in some implementations. The search languageallows construction of logical keyword combinations. The search libraryalso manages the translation of GPS coordinates into place names. Forcolor related searches, color names are converted into RGB value rangesand compared against 9-tile color palettes stored in the image metadata244. Other data to keyword transformations can include, for example,converting dates into company-specific calendar events automaticallyextracted automatically from Facebook Events or manually fromspreadsheets.

Search keywords can be combined with logical operations such as AND, ORand NOT. In some implementations, the language is modeled on the Googlesearch operators and includes a subset of regular expression operatorssuch as * for arbitrary string matching and number ranges specified withellipsis, as in 50 . . . 500. Search begins by breaking the searchstring into logical combinations of keywords. The system then traversesthe image catalog 114 database ranking each of the keywords using ametric specific to the type of keyword search.

Hits on auto-tagged keywords use the confidence value computed duringcategorization, and hits on manually tagged keywords always have ahigher precedence than auto-tagged keywords. Similarly, hits on manuallyentered place names have higher weight than hits on locations generatedfrom GPS coordinates. The keywords are then combined using their logicaloperations to extend or filter the set using standard logicalprecedence. The final set of rankings is processed using a clusteringalgorithm and heuristic thresholds. The top results are returned anddisplayed, typically in rank order unless another ranking metric hasbeen manually applied. In some implementations, the search library canuse suggested search terms.

Still referring to FIG. 6A, the navigator as shown in FIG. 6A has threemain visualization types below the search bar 602: thumbnails 612,graphs, 616, and maps 614. All views are dynamically linked, showing thecurrent search set. Changes in any window are immediately reflected inall windows. Thumbnails 612 display a reduced-sized version of the fullimage, graphs 616 display quantitative information as bar, calendar, orscatter plots, and maps 614 display clustered images and heatmaps.Tapping on regions in each view restricts the current search byfiltering the results based on the tapped location. For example, tappingon a #Dog bar in the category bar graph can filter results to thosecontaining the #Dog tag, or tapping on the search icon 618 in the mapview 614 can filter results to those that are currently visible in themap view. These interactions are converted into standard search stringsand logically combined with the existing search string in order toretain state and allow undo and redo through the search stack.

The thumbnail view 612 is a zoomable flow layout of thumbnails which canbe ordered using a number of different metadata fields. When zoomed out,thumbnails are replaced with simplified palettes to minimize aliasingand improve readability. Flow layouts are a modified grid that variesthe number of images along each row based on aspect ratio to provide amore natural layout. In some implementations, by default, the thumbnailview 612 is ordered by creation date, and grouped into months and yearlysections, depending on the number of images in each section. Orderingcan be based on any metadata field that provides a unique ordering ofimages, including but not limited to, by palette (providing a colorspectrum), modification date, resolution, or camera attributes.

Graphs 616 display summary information for any metadata field, eithercomputed during analysis or embedded by other applications or thecamera, in a variety of useful formats, each of which is interactive,allowing the viewer 124 to point and click to select a subset of imagesquickly and intuitively. The graph view 616 is an extensible set ofinteractive bar, line, scatter and 3D charts that display statisticalinformation about the current set of images. Graphs 616 can be added andorganized to provide a custom dashboard displaying information useful tothe viewer's 124 colleagues or customers. Each graph type can be set touse any appropriate metadata field. For example, the viewer 124 can havea bar chart that shows the frequency of each tag in the viewer's 124portfolio, or it can show the breakdown of images based on aperture orresolution.

Graphs 616 are interactive. Tapping on a graph element can filter thecurrent search results to show the set associated with the tappedregion. For example, as shown in FIG. 6B, tapping on a specific imagetag shows only images that have that tag, while tapping on a specificcolor selects images that contain that color range. Specialized graphtypes exist for common and useful image data. For example, colorspectrum scatter charts show the breakdown of images based on color, andcalendar heatmaps show the frequency of image creation at differenttimes of day, week or year.

The map view 614 shows a map with thumbnail clusters and/or heat mapoverlays. The search results can be filtered to show only images withinthe current visible map region with a single tap. Heat map overlays canbe based off of any image attribute. For example, the viewer 124 cangraph a heat map of the aperture or lens type used to take each image,or use the creation date to see where images have come from over time.The map background can be varied in accordance with someimplementations, selecting either the default native map, graphicalvariations from Stamen and Open Street Maps, or using a proprietary mapserver.

The dashboard 600 as shown in FIG. 6A is configurable. The viewer 124can add or remove graphs or maps of various types to provide acustomized view that the viewer 124 can share with colleagues. Forexample, the viewer 124 can provide a dashboard with graphs showing atime-based calendar view of all the images in the viewer's 124 portfolioalong side a graph of all the categories across all image repositories(e.g., the image databases 123) to make it easy to search in both timeor by category by simply clicking on the graphs. FIG. 6B is an exampleuser interface 622 illustrating keyword search by clicking on thekeyword category bars. The graph view 616 and the map views 614 areinteractive, clicking on them will refine the search. For example, theviewer 124 can restrict the search to the images visible within thecurrent map view by clicking on the search icon 618, or as shown in FIG.6B, by clicking on one of the category bars in a graph. All views arelinked, providing an overview of the entire portfolio or current searchset.

The graphic view 616 and the map view 614 are interactive. Clicking onvarious elements in the dashboard, such as a region on the map 614, abar or wedge in a chart of the graph 616, or a set of thumbnails 612,can update the search and all of the other graphical views immediately.Rather than typing searches in the search area 608, the viewer 124 canfind images with a few taps as further shown in FIG. 6B. The back 604and forward 606 buttons allow the viewer 124 to navigate the searchhistory to quickly try a wide range of searches. The statistical graphs616 and maps 614 show the profile of the current search or the entireimage portfolio, providing an overview of millions of images uploaded tothe image catalog 114.

Still referring to FIG. 6A, the viewer application 102 allows the userto add manual tags to both individual and groups of images (e.g. thecurrent search set) by the categorization module 436. Tags can bedisplayed in the graph tools where they can be used to refine searches.For example, clicking on the #Dog bar in the category graph can selectall images with the #Dog tag. Tags can also be removed from images,including automatically added tags, which can be useful when moving animage through a workflow or when correcting imprecise automaticallyadded tags.

The curator application computes a set of tags automatically duringimage analysis (e.g., by the image analysis module 224 and/or the imageanalysis module 334) as described above, using computer visionalgorithms among others. But these tags can be imprecise and they cannotcapture context-specific information that is not readily available inthe image. For example, they might figure out that it is a picture of ahockey game, but not that it is the final game of the Stanley Cup, orthey might not know that the house in the picture was a user's firstreal estate purchase. Manual tags are always given priority overautomatic tags in search rankings.

Data from the graphs 616 can be exported to spreadsheets to build a costmodel of image use at the viewer's 124 organization. Monitoring imageaccess, storage size and search queries provides the data needed totrack the cost of storing, backing up and using the viewer's 124 imagedatabase to optimize company expenditures and revenue.

In addition to exporting from the graphs 616, the viewer tool candeliver search results and individual images to other applications andpeople in a variety of ways. In some implementations, a SDK provides amechanism for clients to embed custom delivery mechanisms, including theability to interact with proprietary systems. The current search groupcan always be immediately exported with a single click. The user candeliver either the URL to reference the original file location, or fullimages, with options to control naming, resolution, file format, andmetadata stripping and additions. For example, the user can choose todeliver the images as JPG files with a specific resolution, strippingoff all metadata and stamping in copyright and owner information intothe metadata.

Because some of the files may not be directly available if they arestored on offline local storage, the delivery module 440 can deliver allavailable images immediately, and schedule transfer of the remainingfiles once they are available, similar to how a store might ship yourorder in separate packages. Offline delivery relies on the catalogtransfer request protocol as described in further details below.

In some implementations, the viewer desktop application can be developedfor Windows or OSX. Native browser and mobile versions can be added. Forexample, a viewer application can be implemented as a native iOS app,relies on a BaaS (Backend as a Service) database, which uses a customdatabase implementation to optimize for huge image collections.

FIG. 6C illustrates an example map view with a path computed using GPSdata from the smartphone photos to geotag DSLR pictures by date inaccordance with some implementations. In the example map view, theviewer 124 can add location data to images that do not have GPS data bycorrelating images on the map. For example, photos taken with legacycameras, logos, or other generated images can be correlated with GPSdata gathered from smartphones or company events. Trip tracks can bedisplayed on top of the map and interactively dragged to adjust theirexact position. The correlated images are displayed along the track andcan be interactively dragged along the track to adjust their exactposition to assign precise geo location information that can then beused in standard map views. The user has options for making the pathstick to existing roads in the map, and to ignore certain GPS pointscoming from photos that are not desired in the path. The user can modifythe path, and the locations of the images without GPS data are computedon-the-fly.

In some implementations, heatmaps displayed on the map view 614 showlocation based information. FIG. 6D is an example heatmap illustratingthe computation results of the tourists and locals in San Francisco. InFIG. 6D, based on metadata extracted from the images, some images arecategorized as taken by tourists 644, while some images are categorizedas taken by locals 642. Together with geo location informationassociated with each image, a heatmap as shown in FIG. 6D can bedisplayed on the map view 614 to show location based information such asthe spots predominantly visited by tourists as opposed to visited bylocals.

Referring back to FIG. 6A, when a single image is selected from thedashboard, the full metadata is displayed and the original image isdisplayed, if it is available. For locally stored or cloud-based images,the full resolution image is downloaded and cached, and for offlinelocal storage, the full resolution image is requested via a secureasynchronous catalog transfer protocol in accordance with someimplementations. If multiple variations of the same image exist, eitherexact duplicates, crops or color corrected variations of the same image,the system displays those on a timeline. This allows the viewer 124 tosee all the locations where an image is stored, if it has been uploadedto social services (e.g. Facebook), and track the progress through adesign sequence. Variations are detected using SIFT analysis asdisclosed above and other structural image algorithms.

In some implementations, after performing the image analysis anduploading process as shown FIG. 5A, the server 108 receives a query froma second user (e.g., when a single image is selected from the dashboard600 by a viewer 124), matches the query to a first index entry in theimage catalog 114, wherein the first index entry corresponds to a firstimage in the first image database (e.g, the image database 123),determines whether the second user is authorized to view the firstimage, and when the second user is authorized to view the first image,the server 108 retrieves the first image from the first image database(e.g, the image database 123), and transmits the first image to thesecond user. FIG. 7 below illustrates the catalog transfer process inresponse to image requests in further details.

FIG. 7 illustrates a catalog transfer process 700 in accordance withsome implementations. The catalog transfer is handled by modules such asthe transfer module 226 and/or the transfer module 336 to provide amechanism to route requests to the original source of an image, such asthe original images stored in the image databases 123. In someimplementations, the catalog transfer request protocol 700 is anencrypted communication mechanism that supports a broad set ofauthentication protocols (e.g. OAuth, SQL) implemented as pluginmodules. Image specific data, including multiple resolution images (i.e.thumbnails) and EXIF metadata, is requested and delivered to clientapplications after successful authentication. All data is transmittedusing advanced encryption (e.g., 256 bit AES) to prevent unauthorizedinterception. In some implementations, all data stored and transferredto the image catalog 114 is encrypted. Locally cached data is storedunencrypted by default in accordance with some implementations, thoughit can optionally be stored encrypted at the cost of performance. Insome implementations, an industry standard 128 bit encryption, with anoptional 1024 bit encryption, can be used for security. Once the server108 receives the request from the viewer device 104, the server 108notifies (712) the owner, the data manager 132 of the curator device130.

Notifications display changes to the images and requests for action. Forexample, a user can watch a given file or any files that match a givensearch and receive notifications whenever those files are accessed ormodified. The user can also receive notifications when images stored ina local repository on the user's system are requested by remote clients,and finally, notifications can be sent manually to a set of clientsbased on a variety of scoping mechanisms. Both the viewer applicationand the curator application display notifications using the standardmechanisms supported on their native platform 510. In addition to thein-app notification mechanisms, notifications can be sent via email.Emails can include the notification message, and optionally theassociated URL and image files (assuming they are small enough).

One type of notification is a watch notification. In someimplementations, user can request notifications for any search set inthe navigator depicted in FIG. 6A. The search set is “dynamic” in thatit represents any images that match the search criteria, even if newimages are added to the portfolio after the notification watch isrequested. The notification options allow the user to select whethernotification is triggered when any image in the search set is accessed,modified, added or deleted. For example, the user can requestnotification any time any image with the tag #Schnauzer is modified sothat nobody changes any of the pictures of the user's favorite pet.

Another type of notification is a manual notification. In someimplementations, manual notifications can be sent to all people who haveaccess, modified, added or removed images for a given search. This issimilar to a watch notification except that the notification istriggered manually, rather than by any modification to the portfolio.Additionally, the notification is sent to anyone who matches thenotification criteria, not just those people who have watched the imagesin the notification set. This can be useful as a workflow aid, to informpeople when an image has moved from draft to final status, or when theimages in the search set have been posted publicly.

A third type of notification is a transfer notification. In someimplementations, a transfer request is issued any time someone requests(710) access to the pixels in a given image. Transfer requestsautomatically post notifications (712) to the client 132 that manages agiven image, that is, the client 132 that was responsible for uploadingthe requested image metadata via the curator device 130. Transferrequests can be configured to be satisfied automatically via acustomizable rule set, or the user can control access manually byforcing manual approval of the transfer request, providing an extralevel of security.

In some implementations, special tags are used internally by fornotifications. Each user or group that watches a given image is markedwith an @<user> tag. Whenever the notification criteria are satisfied(e.g., if a user is monitoring changes and the image metadata ischanged), the user watching the image is notified.

Still referring to FIG. 7, once the owner 132 approves (714) thetransfer request, the requester 124 can be notified (716) to start theimage download. During the download, the protocol SDK begins with aninitial authentication request that provides access to a given imagerepository. Once authenticated, subsequent requests (718 and 720) useindividual image URLs to retrieve pixel or metadata information. Eachimage can be interrogated to discover the versions and resolutionssupported, and the pixel and metadata information can be requestedseparately for each version. Included modules that implement thetransfer request protocol use the native SDKs for each database type(e.g. Dropbox, Flickr, Filemaker, etc.). In some implementations, thepublic SDK can be used to add support for other proprietary databases.

The system 100 relies on the native image database (e.g., the imagedatabase 123) to fulfill requests for the full-resolution image pixels.Since the native database (e.g., the image database 123) may be local,remote or offline, the assumption is that requests for the full imagepixels may take a long time, even hours or days. In someimplementations, the system tries to fulfill requests for image data,but, if not possible, a special return code can be sent indicating thatthe request may take longer and a notification can be sent when the datais available. After receiving a transfer notification, the pixel datacan then be queried again, with a reasonably, though not guaranteed,assumption that the pixels can be returned in a few seconds. However, ifthe notification is not serviced within a specified time period (hours),the requester 124 may receive another notification in the pixel resultblock.

In some implementations, transfer requests are serviced immediately forlocal and online cloud storage (depending on security options). Offlinestorage requests are buffered and stored in the image catalog 114. Onceoffline storage becomes available, e.g. when the desktop attached to thestorage goes online running either the curator or viewer application, anotification is sent (716) to the data manager 132 of the curator device130, which then transfers (718) the pixels to the image catalog 114,where it is stored in a temporary container 702. Once the pixels aretransferred (718), a second notification is sent to the originalrequester 124, who can then download (720) the pixels to the viewerdevice 104. If the original requestor 124 is not online, the secondnotification is buffered and re-sent once the original requestor 124comes back online in accordance with some implementation. Once thepixels are downloaded by the original requestor 124, or if a request hasbeen outstanding for more than a specified period (typically a fewhours), the temporary container 702 residing on the image catalog 114 isreleased and the transfer requests must be performed again as needed.

The long latency of transfer requests demands special procedures inapplications using the image catalog 114. They are prepared to usestand-in thumbnails and never expect to have fast access to thefull-resolution image. The advantage is that the system can work withimages that are offline or when the network is down, removing therequirement that all image databases be online and accessible for theapp to run properly. Transfer requests thus enable the system to handlemultiple databases (e.g., the image databases 123) with high security,including offline storage attached to local computing devices as opposedto high-reliability cloud services.

The curator and viewer applications described herein leverage a numberof interrelated libraries to improve search. The libraries are useddirectly by the desktop and mobile application and a set of SDK modules,or can be accessed remotely via a REST API running on a server managingthe image catalog 114 by web-based JavaScript apps in accordance withsome implementations. In some implementations, all of the functions areasynchronous, returning results using a non-blocking API designed tosupport multi-threaded clients accessing data over the Internet withhigh latency. In some implementations, the set of SDK modules, PublicSDK, described in more details below, can be used by customers todirectly to extend the image catalog.

The image catalog 114 along with the modules to access and manage theimage catalog 114 provides a management layer that makes multiple localor remote image repositories (e.g., the image databases 123) appear as asingle, asynchronous repository. The image analysis library provides acollection of advanced image processing that can be used to extractinformation from images useful for search. The extracted information canbe stored in the metadata 244. To facilitate the image search, the imagesearch library further ranks and filters images managed by the imagecatalog 114.

Typical cloud databases assume that all the data is stored in a singlecontainer, accessible through a single authentication. Requiring asingle repository means changing a user's workflow, creating duplicatecopies of data, and increasing storage costs. For enterprise customerswith a large collection of images (e.g., the Smithsonian Institution ora professional sports team) such workflow changes are disruptive.

In comparison, the disclosed implementations address these problems byconstructing a lightweight database, orders of magnitude smaller than atypical image database, containing placeholders for each image, muchlike a card catalog does for a library. This lightweight database isdecoupled from the geometrically increasing resolution and file size ofimage and video data (e.g., book contents), enabling it to scale tomassive image collections at a very low storage cost (e.g., a smallcabinet). Additionally, images may live in any location, allowing theuser to group collections of images stored in separate repositories,avoiding duplication and workflow changes.

Disclosed implementations are image-centric. Images are a privilegedfile type on the Internet. Many social and cloud services treat imagesdifferently than generic files. They can be displayed, shared withothers, and stored in many places that do not permit generic files.Images can contain metadata, which includes hidden text stored withinthe file that is not visible without specialized tools. Metadata isextensible and can contain any information useful to the app thatcreated the image. Images are also resizable, with standard tools thatprovide smaller representations, thumbnails, of larger images. Imagesare typically quite large and their file sizes are growinggeometrically. Cameras routinely generate 40-60 megapixel images whichcan easily take up 100 MB of storage space. Some implementationsdecouple these storage size issues by leveraging the image metadata andthumbnails to provide ways of transferring images around networksefficiently.

In some implementations, the Public SDK provides interfaces to extendthe image catalog platform. Modules are provided to add new databasetypes to the curator tool, export search results to externalapplications, provide domain-specific data and computed metadata fieldsto use during search. In some implementations, extension modules arewritten in the C/C++ and run inside of the curator and viewerapplications. In some implementations, the Public SDK includes but notlimited to a Portfolio SDK, an Export SDK, and a Domain Data SDK.

The Portfolio SDK enables addition of new image sources. Image sourcemodules 222 and/or 332 provide authentication, inventory, access andmodification methods that are used by the curator and viewerapplication. In some implementations, the Portfolio SDK provides curatormodules for Flickr, Facebook, Google+, Dropbox, Box and many others.Custom portfolio modules can be added to access images stored inproprietary repositories. In order to access the images stored in sourceimage repositories (e.g., images stored in image databases 123 onexternal services 122 such as Flickr, Facebook, Google+, Dropbox, Boxand many others), the Portfolio SDK can include:

Overrides:    Open (URL)    Search(string, A(string, GUID[ ]))   GetThumbnail(GUID[ ], {circumflex over ( )}(GUID[ ], Thumbnail))   GetMetadataField(GUID[ ], field, {circumflex over ( )}(GUID[ ],field, value[ ])    SetMetadataField(GUID, field, value, {circumflexover ( )}(GUID, field, value, status))    RequestPixels(GUID,{circumflex over ( )}(GUID, pixels))

The Export SDK connects search results to external applications. It isinvoked by the viewer application (e.g., the delivery module 440) todeliver image groups to a variety of services. In some implementations,the present disclosure provides export modules for LightroomCollections, email, and Wordpress. The Export SDK can include:

Overrides:    Init( )    Export(Portfolio, GUTD[ ])    Finish( )

The Domain Data SDK converts proprietary files and spreadsheets intoglobal or per-image metadata fields to aid in searching. For example,the user can import company calendars that identify events, orspreadsheets that contain per-image data not included in the originalthumbnails. In some implementations, if the user has a custom databasethat contains image data, through Domain Data SDK, the user can parseand extract that data and add it to the metadata 244 stored in the imagecatalog 114.

In some implementations, global data is also used for suggestedcompletions and tag selections. For example, if a department uses aspecific workflow based around specific tag sets, or if the user likesto group the user's images into specific tag sets that may not exist onany image, the user can define global tags. In another example, the usercan define a tag sets called “Workflow” with the tags {#Draft, #Eval,#Final}, or “Seasons” with {#Winter, #Spring, #Summer, #Fall}. TheDomain Data SDK can include:

Overrides:    Init( )    bool Open(Portfolio, URL) Finish( ) Mutators:   Register(“.ext; .typ”)    SetGlobal (name, value)    MakeTag(name)

FIGS. 8A-8B illustrate a flowchart of a process 800 for managing animage catalog. In some implementations, the process 800 is performed byone or more servers (e.g., the server system 108, FIG. 1), each havingone or more processors and memory.

The server system 108 receives (802) from a first user (e.g., the datamanager 132, FIG. 1 and FIG. 5A) identification of one or more images ina first image database (e.g., the image database 123, FIG. 1). In someimplementations, the first image database is distinct from the one ormore servers. For example, as shown in FIG. 1, the image databases 123can reside on one or more external services 122 that are distinct fromthe server system 108.

For each image of the one or more images (804), the server system 108first analyzes (806) the respective image to extract respective keywordsthat describe the respective image. In some implementations, analyzingthe respective image to extract respective keywords includes (808)performing deep convolutional neural network semantic analysis on therespective image. For example, as shown in FIG. 5C, an image of awaterfront sunset scene with clouds in the sky can be analyzed byperforming deep convolutional neural network semantic analysis on theimage, and keywords such as “sunset,” “cloud,” and “water” describingthe image can be extracted and saved as tags.

In some implementations, the first user specifies (810) a subject matterfor the one or more identified images, and the deep convolutional neuralnetwork semantic analysis uses a neural network trained on imagescorresponding to the specified subject matter. For example, during atraining phase of the deep convolutional neural network semanticanalysis, a user supplies images with sunset scenes and a subject matterkeyword “sunset” to the deep convolutional neural network. The trainedneural network can be then used in the deep convolutional neural networksemantic analysis to recognize images with sunset scenes and producekeywords such as “sunset” as tags for the images.

As shown in FIG. 5A, in addition to or in conjunction with the deepconvolutional neural network semantic analysis, other image analysistechniques such as OCR, color analysis, and facial and body recognitionare also used to analyze the respective image and generate respectivetags. In some implementations, analyzing the respective image to extractrespective keywords includes (812) using optical character recognition(OCR), e.g., using OCR to extra logos, street signs, jersey numbers, ordocument text. In some implementations, analyzing the respective imageto extract respective keywords also includes (814) extracting a colorpalette for the respective image, as illustrated in FIGS. 5D and 5E. Insome implementations, analyzing the respective image to extractrespective keywords includes (816) identifying one or more known facesin the respective image. In some implementations, analyzing therespective image to extract respective keywords includes identifying(818) one or more known human bodies based on body features (e.g., matchfacial and body features against a set of known geometries to generatenames).

After analyzing the respective image, the server system 108 creates(820) a respective index entry in the image catalog, where therespective index entry includes the respective keywords, such as“sunset.” The indexed image catalog can then be searched. In someimplementations, through a user interface (e.g., the exemplary imageportfolio dashboard as shown in FIG. 6A), the server system 108 receives(822) a query from a second user (e.g., the requester 124, FIG. 1 andFIG. 7). Upon receiving the query, the server system 108 matches (824)the query to a first index entry in the image catalog, where the firstindex entry corresponds to a first image in the first image database.

In some implementations, security is built into the process 800 asdescribed above with respect to FIGS. 5 and 7. During catalog creation,an approval process is in place to allow only authorized uploads, andbefore returning search results to a requestor, the server system 108authenticates the requestor and provides search results only toauthorized users. In some implementations, upon receiving a searchresult, the server system 108 determines (826) whether the second user(e.g., the requester 124, FIGS. 1 and 7) is authorized to view the firstimage. When the second user is authorized to view the first image (828),the server system 108 retrieves (830) the first image from the firstimage database, and transmits (832) the first image to the second user.In some implementations, some of the files may not be immediatelyavailable upon request. For example, a file may be stored in offlinelocal storage or by external services 122 that are briefly unavailable.When the requested files are not immediately available, the serversystem 108 can queue the request and schedule transfer of the files oncethey are available, such as when an offline storage medium or anexternal service 122 is back online.

In some implementations, in addition to the storing tags generated byimage analysis, other metadata is also stored in the image catalog tofacilitate searches. In some implementations, for each image (834), theserver system 108 identifies (836) metadata associated the respectiveimage. The metadata may include: a date/time the respective image wascreated; a location where the image was created; identification of acamera that took the respective image; and/or identification of cameraattributes that took the respective image. The server system 108 stores(838) the metadata as part of the index entry for the respective image.This metadata can be used when displaying the image portfolio to a useror during a search. For example, the location where the image wascreated (e.g., GPS data) can be stored as metadata and/or as part of thelocation index entry for the image. During search, the location metadatacan be used to generate a map view of images with a path computed usingGPS data, as shown in FIG. 6C, or generate a heatmap to show popularspots where tourists take photos as opposed to where locals take photos,as shown in FIG. 6D.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theimplementations were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious implementations with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method of managing an image catalog, performedby one or more servers, each having one or more processors and memory,the method comprising, at the one or more servers: receivingreduced-resolution versions of one or more images stored in a firstimage database of an external service distinct from the one or moreservers; for each of the received reduced-resolution versions, creatinga respective index entry in the image catalog; receiving a query from auser; matching the query to a first index entry in the image catalog,the first index entry corresponding to a first image stored as afull-resolution version in the first image database; transmitting, to anowner of the first image, a request for authorization of the user toview the first image; and in accordance with a determination that theuser is authorized to view the first image: retrieving thefull-resolution version of the first image from the first image databaseof the external service; temporarily storing the full-resolution versionof the first image in a temporary storage of the one or more servers;and transmitting the full-resolution version of the first image to theuser; and releasing the full-resolution version of the first image fromthe temporary storage in response to the transmitting of thefull-resolution version of the first image to the user.
 2. The method ofclaim 1, wherein the respective index entry includes one or morekeywords extracted from the respective reduced-resolution version. 3.The method of claim 2, wherein the one or more keywords are extractedfrom the reduced-resolution versions by performing deep convolutionalneural network semantic analysis on the respective reduced-resolutionversion.
 4. The method of claim 3, further comprising receiving aspecified subject matter for the one or more images, wherein the deepconvolutional neural network semantic analysis uses a neural networktrained on images corresponding to the specified subject matter.
 5. Themethod of claim 2, wherein the one or more keywords comprise a colorpalette for the respective reduced-resolution version.
 6. The method ofclaim 2, wherein the one or more keywords comprise one or more knownfaces in the respective reduced-resolution version.
 7. The method ofclaim 1, further comprising for each respective image: identifyingmetadata associated the respective reduced-resolution version, whereinthe respective metadata is selected from the group consisting of date ortime when the respective image was created, location where therespective image was created, identification of a camera that took therespective image, and identification of camera attributes that took therespective image; and storing the respective metadata as part of therespective index entry for the respective reduced-resolution version. 8.A computer system for managing an image catalog, comprising: one or moreprocessors; memory; and one or more programs stored in the memory andconfigured for execution by the one or more processors, the one or moreprograms comprising instructions for, at the computer system: receivingreduced-resolution versions of one or more images stored in a firstimage database of an external service distinct from the computer system;for each of the received reduced-resolution versions, creating arespective index entry in the image catalog; receiving a query from auser; matching the query to a first index entry in the image catalog,the first index entry corresponding to a first image stored as afull-resolution version in the first image database; transmitting, to anowner of the first image, a request for authorization of the user toview the first image; and in accordance with a determination that theuser is authorized to view the first image: retrieving thefull-resolution version of the first image from the first image databaseof the external service; temporarily storing the full-resolution versionof the first image in a temporary storage of the computer system; andtransmitting the full-resolution version of the first image to the user;and releasing the full-resolution version of the first image from thetemporary storage in response to the transmitting of the full-resolutionversion of the first image to the user.
 9. The computer system of claim8, wherein the respective index entry includes one or more keywordsextracted from the respective reduced-resolution version.
 10. Thecomputer system of claim 9, wherein the one or more keywords areextracted from the reduced-resolution versions by performing deepconvolutional neural network semantic analysis on the respectivereduced-resolution version.
 11. The computer system of claim 10, whereinthe one or more programs further comprise instructions for receiving aspecified subject matter for the one or more images, and the deepconvolutional neural network semantic analysis uses a neural networktrained on images corresponding to the specified subject matter.
 12. Thecomputer system of claim 9, wherein the one or more keywords comprise acolor palette for the respective reduced-resolution version.
 13. Thecomputer system of claim 9, wherein the one or more keywords compriseone or more known faces in the respective reduced-resolution version.14. The computer system of claim 8, wherein the one or more programsfurther comprise instructions that execute for each respective image,for: identifying metadata associated the respective reduced-resolutionversion, wherein the respective metadata is selected from the groupconsisting of date or time when the respective image was created,location where the respective image was created, identification of acamera that took the respective image, and identification of cameraattributes that took the respective image; and storing the respectivemetadata as part of the respective index entry for the respectivereduced-resolution version.
 15. A non-transitory computer readablestorage medium storing one or more programs configured for execution byone or more processors of a computer system for managing an imagecatalog, the one or more programs comprising instructions for: receivingreduced-resolution versions of one or more images stored in a firstimage database of an external service distinct from the computer system;for each of the received reduced-resolution versions, creating arespective index entry in the image catalog; receiving a query from auser; matching the query to a first index entry in the image catalog,the first index entry corresponding to a first image stored as afull-resolution version in the first image database; transmitting, to anowner of the first image, a request for authorization of the user toview the first image; and in accordance with a determination that theuser is authorized to view the first image: retrieving thefull-resolution version of the first image from the first image databaseof the external service; temporarily storing the full-resolution versionof the first image in a temporary storage of the computer system; andtransmitting the full-resolution version of the first image to the user;and releasing the full-resolution version of the first image from thetemporary storage in response to the transmitting of the full-resolutionversion of the first image to the user.
 16. The computer readablestorage medium of claim 15, wherein the respective index entry includesone or more keywords extracted from the respective reduced-resolutionversion.
 17. The computer readable storage medium of claim 16, whereinthe one or more keywords are extracted from the reduced-resolutionversions by performing deep convolutional neural network semanticanalysis on the respective reduced-resolution version.
 18. The computerreadable storage medium of claim 17, wherein the one or more programsfurther comprise instructions for receiving a specified subject matterfor the one or more images, and the deep convolutional neural networksemantic analysis uses a neural network trained on images correspondingto the specified subject matter.
 19. The computer readable storagemedium of claim 16, wherein the one or more keywords comprise a colorpalette for the respective reduced-resolution version.
 20. The computerreadable storage medium of claim 15, wherein the one or more programsfurther comprise instructions that execute for each respective image,for: identifying metadata associated the respective reduced-resolutionversion, wherein the respective metadata is selected from the groupconsisting of date or time when the respective image was created,location where the respective image was created, identification of acamera that took the respective image, and identification of cameraattributes that took the respective image; and storing the respectivemetadata as part of the respective index entry for the respectivereduced-resolution version.